{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Training OpenAI GPT-2-CH08.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LH2YgC7LfzJZ",
        "colab_type": "text"
      },
      "source": [
        "#Training OpenAI GTP-2\n",
        "Copyright 2020, Denis Rothman MIT License. Denis Rothman created the Colab notebook using the OpenAI repository, adding title steps for educational purposes only.\n",
        "\n",
        "***Code References***\n",
        "\n",
        "[Reference: OpenAI Repository](https://github.com/openai/gpt-2)\n",
        "The repository was cloned and adapted to N Shepperd's repository.\n",
        "\n",
        "[Reference: N Shepperd Repository](https://github.com/nshepperd/gpt-2)\n",
        "The repository was not cloned. N Shepperd's training programs were inserted into the OpenAI Repository. The list of N Shepperd's programs are cited in the 'N Shepperd' section of the notebook. Some programs were modified for educational purposes only to work with this notebook.\n",
        "\n",
        "***Model Reference Paper***\n",
        "\n",
        "[Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever,2019,'Language Models are Unsupervised Multitask Learners'](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf)\n",
        "\n",
        "\n",
        "***Step 1: Pre-requisites:***\n",
        "\n",
        "a) activate GPU in the notebook settings runTime menu <br>\n",
        "b) Upload the following program files and dset.txt(dataset) with the file manager: train.py,load_dataset.py,encode.py,accumulate,memory_saving_gradients.py,dset.txt"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "isqdu1fpfmqM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "0662d019-7248-4642-c840-7b87c08e7ce7"
      },
      "source": [
        "#@title Step 2: Cloning the OpenAI GPT-2 Repository \n",
        "#!git clone https://github.com/nshepperd/gpt-2.git\n",
        "!git clone https://github.com/openai/gpt-2.git"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'gpt-2'...\n",
            "remote: Enumerating objects: 230, done.\u001b[K\n",
            "remote: Total 230 (delta 0), reused 0 (delta 0), pack-reused 230\u001b[K\n",
            "Receiving objects: 100% (230/230), 4.38 MiB | 7.37 MiB/s, done.\n",
            "Resolving deltas: 100% (119/119), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7RHOjN-TjUbj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 887
        },
        "outputId": "cc45d116-e7a5-4ff8-e41b-7d440317c9a8"
      },
      "source": [
        "#@title Step 3: Installing the requirements\n",
        "import os                     # when the VM restarts import os necessary\n",
        "os.chdir(\"/content/gpt-2\")    \n",
        "!pip3 install -r requirements.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fire>=0.1.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/34/a7/0e22e70778aca01a52b9c899d9c145c6396d7b613719cd63db97ffa13f2f/fire-0.3.1.tar.gz (81kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 2.5MB/s \n",
            "\u001b[?25hCollecting regex==2017.4.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/36/62/c0c0d762ffd4ffaf39f372eb8561b8d491a11ace5a7884610424a8b40f95/regex-2017.04.05.tar.gz (601kB)\n",
            "\u001b[K     |████████████████████████████████| 604kB 8.9MB/s \n",
            "\u001b[?25hCollecting requests==2.21.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 6.5MB/s \n",
            "\u001b[?25hCollecting tqdm==4.31.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/4b/c38b5144cf167c4f52288517436ccafefe9dc01b8d1c190e18a6b154cd4a/tqdm-4.31.1-py2.py3-none-any.whl (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 5.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from fire>=0.1.3->-r requirements.txt (line 1)) (1.12.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from fire>=0.1.3->-r requirements.txt (line 1)) (1.1.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests==2.21.0->-r requirements.txt (line 3)) (2020.6.20)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests==2.21.0->-r requirements.txt (line 3)) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests==2.21.0->-r requirements.txt (line 3)) (1.24.3)\n",
            "Collecting idna<2.9,>=2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 6.0MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: fire, regex\n",
            "  Building wheel for fire (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fire: filename=fire-0.3.1-py2.py3-none-any.whl size=111005 sha256=3310fe2adb427d9c42d252d7a50303321e9db5a10c95bd0083efc4df204f9703\n",
            "  Stored in directory: /root/.cache/pip/wheels/c1/61/df/768b03527bf006b546dce284eb4249b185669e65afc5fbb2ac\n",
            "  Building wheel for regex (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for regex: filename=regex-2017.4.5-cp36-cp36m-linux_x86_64.whl size=533204 sha256=410a1a2649a21cad83bbd2d67acd95e54704541f49ca03c2ac08574a44ff5985\n",
            "  Stored in directory: /root/.cache/pip/wheels/75/07/38/3c16b529d50cb4e0cd3dbc7b75cece8a09c132692c74450b01\n",
            "Successfully built fire regex\n",
            "\u001b[31mERROR: spacy 2.2.4 has requirement tqdm<5.0.0,>=4.38.0, but you'll have tqdm 4.31.1 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: google-colab 1.0.0 has requirement requests~=2.23.0, but you'll have requests 2.21.0 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Installing collected packages: fire, regex, idna, requests, tqdm\n",
            "  Found existing installation: regex 2019.12.20\n",
            "    Uninstalling regex-2019.12.20:\n",
            "      Successfully uninstalled regex-2019.12.20\n",
            "  Found existing installation: idna 2.9\n",
            "    Uninstalling idna-2.9:\n",
            "      Successfully uninstalled idna-2.9\n",
            "  Found existing installation: requests 2.23.0\n",
            "    Uninstalling requests-2.23.0:\n",
            "      Successfully uninstalled requests-2.23.0\n",
            "  Found existing installation: tqdm 4.41.1\n",
            "    Uninstalling tqdm-4.41.1:\n",
            "      Successfully uninstalled tqdm-4.41.1\n",
            "Successfully installed fire-0.3.1 idna-2.8 regex-2017.4.5 requests-2.21.0 tqdm-4.31.1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "idna",
                  "requests",
                  "tqdm"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q9vV73Opw68m",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "8d3e336b-7385-4a51-f054-bf3a1ffd3b6a"
      },
      "source": [
        "!pip install toposort"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting toposort\n",
            "  Downloading https://files.pythonhosted.org/packages/e9/8a/321cd8ea5f4a22a06e3ba30ef31ec33bea11a3443eeb1d89807640ee6ed4/toposort-1.5-py2.py3-none-any.whl\n",
            "Installing collected packages: toposort\n",
            "Successfully installed toposort-1.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_kpNCnh9fyYD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "6915ef8b-a48f-4a27-c6d4-43fda10b0e82"
      },
      "source": [
        "#@title Step 4: Checking TensorFlow version \n",
        "#Colab has tf 1.x and tf 2.x installed\n",
        "#Restart runtime using 'Runtime' -> 'Restart runtime...'\n",
        "%tensorflow_version 1.x\n",
        "import tensorflow as tf\n",
        "print(tf.__version__)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "1.15.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jvVj0cLVkaPL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "12f91649-5661-4323-887a-bed1456ce370"
      },
      "source": [
        "#@title Step 5: Downloading 117M parameter GPT-2 Model\n",
        "# run code and send argument\n",
        "import os # after runtime is restarted\n",
        "os.chdir(\"/content/gpt-2\")\n",
        "!python3 download_model.py '117M' #creates model directory"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rFetching checkpoint:   0%|                                              | 0.00/77.0 [00:00<?, ?it/s]\rFetching checkpoint: 1.00kit [00:00, 781kit/s]                                                      \n",
            "\rFetching encoder.json:   0%|                                           | 0.00/1.04M [00:00<?, ?it/s]\rFetching encoder.json: 1.04Mit [00:00, 34.0Mit/s]                                                   \n",
            "Fetching hparams.json: 1.00kit [00:00, 1.04Mit/s]                                                   \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:10, 47.8Mit/s]                                  \n",
            "Fetching model.ckpt.index: 6.00kit [00:00, 4.75Mit/s]                                               \n",
            "Fetching model.ckpt.meta: 472kit [00:00, 35.2Mit/s]                                                 \n",
            "Fetching vocab.bpe: 457kit [00:00, 35.1Mit/s]                                                       \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aV5K8rvD1b-r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Step 6: Copying the Project Resources to scr\n",
        "!cp /content/mdset.txt /content/gpt-2/src/\n",
        "!cp -r /content/gpt-2/models/ /content/gpt-2/src/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dTUxDwtWlOLf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Step 7: Copying the N Shepperd Training Files\n",
        "#Referfence GitHub repository: https://github.com/nshepperd/gpt-2\n",
        "import os # import after runtime is restarted\n",
        "!cp /content/train.py /content/gpt-2/src/\n",
        "!cp /content/load_dataset.py /content/gpt-2/src/\n",
        "!cp /content/encode.py /content/gpt-2/src/\n",
        "!cp /content/accumulate.py /content/gpt-2/src/\n",
        "!cp /content/memory_saving_gradients.py /content/gpt-2/src/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B6T2OrWoOvG0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "cb7f7b63-837e-4d19-b594-c73a535492c3"
      },
      "source": [
        "#@title Step 8:Encoding dataset\n",
        "import os # import after runtime is restarted\n",
        "os.chdir(\"/content/gpt-2/src/\")\n",
        "model_name=\"117M\"\n",
        "!python /content/gpt-2/src/encode.py mdset.txt out.npz "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading files\n",
            "100% 1/1 [00:00<00:00,  3.98it/s]\n",
            "Writing out.npz\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UzlkNGbAkDBk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "5cbf0d29-6d2d-4630-9c30-07930c37e6dd"
      },
      "source": [
        "#@title Step 9:Training the Model\n",
        "#Model saved after 1000 steps\n",
        "import os # import after runtime is restarted\n",
        "os.chdir(\"/content/gpt-2/src/\")\n",
        "!python train.py --dataset out.npz"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/memory_saving_gradients.py:13: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:89: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:92: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
            "\n",
            "2020-06-29 09:16:57.805692: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299995000 Hz\n",
            "2020-06-29 09:16:57.806095: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2c1b2c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
            "2020-06-29 09:16:57.806138: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
            "2020-06-29 09:16:57.812896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
            "2020-06-29 09:16:57.997614: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:16:57.998629: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2c1b480 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
            "2020-06-29 09:16:57.998671: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
            "2020-06-29 09:16:58.000243: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:16:58.001082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2020-06-29 09:16:58.001644: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-06-29 09:16:58.333957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2020-06-29 09:16:58.555071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2020-06-29 09:16:58.593295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2020-06-29 09:16:58.915089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2020-06-29 09:16:58.956129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2020-06-29 09:16:59.738702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2020-06-29 09:16:59.738968: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:16:59.740099: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:16:59.740924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2020-06-29 09:16:59.745598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-06-29 09:16:59.747346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2020-06-29 09:16:59.747386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 \n",
            "2020-06-29 09:16:59.747406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N \n",
            "2020-06-29 09:16:59.749272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:16:59.750200: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:16:59.750941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10805 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "WARNING:tensorflow:From train.py:93: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:148: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:152: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:36: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:51: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:64: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:16: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:67: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.random.categorical` instead.\n",
            "WARNING:tensorflow:From train.py:118: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:122: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:145: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:148: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:150: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:153: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n",
            "\n",
            "WARNING:tensorflow:From train.py:157: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.\n",
            "\n",
            "Loading checkpoint models/117M/model.ckpt\n",
            "Loading dataset...\n",
            "100% 1/1 [00:00<00:00, 260.74it/s]\n",
            "dataset has 29379 tokens\n",
            "Training...\n",
            "2020-06-29 09:17:29.007668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "[1 | 7.03] loss=3.18 avg=3.18\n",
            "[2 | 7.96] loss=2.67 avg=2.92\n",
            "[3 | 8.90] loss=2.92 avg=2.92\n",
            "[4 | 9.82] loss=3.00 avg=2.94\n",
            "[5 | 10.76] loss=2.65 avg=2.88\n",
            "[6 | 11.69] loss=2.88 avg=2.88\n",
            "[7 | 12.63] loss=2.80 avg=2.87\n",
            "[8 | 13.57] loss=2.68 avg=2.84\n",
            "[9 | 14.52] loss=2.88 avg=2.85\n",
            "[10 | 15.46] loss=3.93 avg=2.96\n",
            "[11 | 16.40] loss=3.06 avg=2.97\n",
            "[12 | 17.34] loss=2.48 avg=2.93\n",
            "[13 | 18.28] loss=2.69 avg=2.91\n",
            "[14 | 19.22] loss=3.19 avg=2.93\n",
            "[15 | 20.16] loss=2.29 avg=2.88\n",
            "[16 | 21.11] loss=2.28 avg=2.84\n",
            "[17 | 22.04] loss=2.91 avg=2.85\n",
            "[18 | 22.97] loss=2.67 avg=2.84\n",
            "[19 | 23.91] loss=2.14 avg=2.80\n",
            "[20 | 24.85] loss=2.00 avg=2.75\n",
            "[21 | 25.78] loss=2.58 avg=2.75\n",
            "[22 | 26.73] loss=2.66 avg=2.74\n",
            "[23 | 27.67] loss=2.80 avg=2.74\n",
            "[24 | 28.60] loss=3.18 avg=2.76\n",
            "[25 | 29.54] loss=2.95 avg=2.77\n",
            "[26 | 30.47] loss=3.41 avg=2.80\n",
            "[27 | 31.41] loss=2.92 avg=2.81\n",
            "[28 | 32.34] loss=2.33 avg=2.79\n",
            "[29 | 33.27] loss=2.17 avg=2.76\n",
            "[30 | 34.20] loss=1.87 avg=2.73\n",
            "[31 | 35.13] loss=2.60 avg=2.72\n",
            "[32 | 36.06] loss=2.71 avg=2.72\n",
            "[33 | 37.00] loss=2.82 avg=2.73\n",
            "[34 | 37.95] loss=2.26 avg=2.71\n",
            "[35 | 38.89] loss=2.20 avg=2.69\n",
            "[36 | 39.83] loss=2.48 avg=2.69\n",
            "[37 | 40.76] loss=2.03 avg=2.66\n",
            "[38 | 41.70] loss=2.15 avg=2.65\n",
            "[39 | 42.64] loss=2.57 avg=2.65\n",
            "[40 | 43.57] loss=2.42 avg=2.64\n",
            "[41 | 44.50] loss=2.20 avg=2.63\n",
            "[42 | 45.43] loss=3.01 avg=2.64\n",
            "[43 | 46.37] loss=2.74 avg=2.64\n",
            "[44 | 47.30] loss=3.33 avg=2.66\n",
            "[45 | 48.24] loss=3.14 avg=2.67\n",
            "[46 | 49.17] loss=2.40 avg=2.67\n",
            "[47 | 50.11] loss=2.58 avg=2.66\n",
            "[48 | 51.04] loss=1.93 avg=2.64\n",
            "[49 | 51.97] loss=3.22 avg=2.66\n",
            "[50 | 52.91] loss=2.56 avg=2.66\n",
            "[51 | 53.84] loss=1.95 avg=2.64\n",
            "[52 | 54.77] loss=2.18 avg=2.63\n",
            "[53 | 55.70] loss=2.65 avg=2.63\n",
            "[54 | 56.63] loss=2.29 avg=2.62\n",
            "[55 | 57.55] loss=2.21 avg=2.61\n",
            "[56 | 58.49] loss=1.98 avg=2.60\n",
            "[57 | 59.41] loss=2.47 avg=2.59\n",
            "[58 | 60.34] loss=1.95 avg=2.58\n",
            "[59 | 61.26] loss=2.40 avg=2.57\n",
            "[60 | 62.19] loss=2.22 avg=2.57\n",
            "[61 | 63.12] loss=3.16 avg=2.58\n",
            "[62 | 64.05] loss=2.25 avg=2.57\n",
            "[63 | 64.99] loss=3.32 avg=2.59\n",
            "[64 | 65.93] loss=2.44 avg=2.59\n",
            "[65 | 66.86] loss=2.39 avg=2.58\n",
            "[66 | 67.79] loss=2.23 avg=2.57\n",
            "[67 | 68.73] loss=2.21 avg=2.57\n",
            "[68 | 69.66] loss=2.45 avg=2.56\n",
            "[69 | 70.58] loss=3.28 avg=2.58\n",
            "[70 | 71.52] loss=2.22 avg=2.57\n",
            "[71 | 72.45] loss=1.76 avg=2.56\n",
            "[72 | 73.38] loss=3.01 avg=2.56\n",
            "[73 | 74.31] loss=2.04 avg=2.55\n",
            "[74 | 75.25] loss=2.20 avg=2.55\n",
            "[75 | 76.18] loss=2.43 avg=2.54\n",
            "[76 | 77.10] loss=3.45 avg=2.56\n",
            "[77 | 78.03] loss=2.40 avg=2.56\n",
            "[78 | 78.96] loss=2.34 avg=2.55\n",
            "[79 | 79.89] loss=2.09 avg=2.55\n",
            "[80 | 80.82] loss=2.17 avg=2.54\n",
            "[81 | 81.75] loss=2.27 avg=2.53\n",
            "[82 | 82.69] loss=2.17 avg=2.53\n",
            "[83 | 83.62] loss=2.19 avg=2.52\n",
            "[84 | 84.56] loss=2.73 avg=2.53\n",
            "[85 | 85.49] loss=2.96 avg=2.53\n",
            "[86 | 86.43] loss=2.20 avg=2.53\n",
            "[87 | 87.37] loss=2.10 avg=2.52\n",
            "[88 | 88.31] loss=2.91 avg=2.53\n",
            "[89 | 89.24] loss=2.91 avg=2.53\n",
            "[90 | 90.17] loss=2.07 avg=2.53\n",
            "[91 | 91.10] loss=2.84 avg=2.53\n",
            "[92 | 92.03] loss=1.77 avg=2.52\n",
            "[93 | 92.96] loss=2.68 avg=2.52\n",
            "[94 | 93.88] loss=2.36 avg=2.52\n",
            "[95 | 94.81] loss=2.65 avg=2.52\n",
            "[96 | 95.74] loss=1.89 avg=2.51\n",
            "[97 | 96.68] loss=2.37 avg=2.51\n",
            "[98 | 97.60] loss=1.99 avg=2.50\n",
            "[99 | 98.53] loss=2.62 avg=2.50\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            "ive, and the two are not related.\n",
            "\n",
            "The second is the same as the first, but the two are not related.\n",
            "\n",
            "The third is the same as the first, but the two are not related.\n",
            "\n",
            "The fourth is the same as the first, but the two are not related.\n",
            "\n",
            "The fifth is the same as the first, but the two are not related.\n",
            "\n",
            "The sixth is the same as the first, but the two are not related.\n",
            "\n",
            "The seventh is the same as the first, but the two are not related.\n",
            "\n",
            "The eighth is the same as the first, but the two are not related.\n",
            "\n",
            "The ninth is the same as the first, but the two are not related.\n",
            "\n",
            "The tenth is the same as the first, but the two are not related.\n",
            "\n",
            "The eleventh is the same as the first, but the two are not related.\n",
            "\n",
            "The twelfth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the two are not related.\n",
            "\n",
            "The thirteenth is the same as the first, but the\n",
            "\n",
            "[100 | 121.68] loss=1.76 avg=2.49\n",
            "[101 | 122.61] loss=2.08 avg=2.48\n",
            "[102 | 123.55] loss=2.05 avg=2.48\n",
            "[103 | 124.49] loss=2.38 avg=2.48\n",
            "[104 | 125.43] loss=2.39 avg=2.47\n",
            "[105 | 126.36] loss=2.23 avg=2.47\n",
            "[106 | 127.31] loss=2.02 avg=2.46\n",
            "[107 | 128.24] loss=2.95 avg=2.47\n",
            "[108 | 129.17] loss=1.90 avg=2.46\n",
            "[109 | 130.11] loss=2.49 avg=2.46\n",
            "[110 | 131.04] loss=2.15 avg=2.46\n",
            "[111 | 131.97] loss=2.17 avg=2.45\n",
            "[112 | 132.90] loss=2.15 avg=2.45\n",
            "[113 | 133.83] loss=2.10 avg=2.44\n",
            "[114 | 134.75] loss=2.61 avg=2.45\n",
            "[115 | 135.68] loss=2.62 avg=2.45\n",
            "[116 | 136.61] loss=2.28 avg=2.45\n",
            "[117 | 137.54] loss=2.04 avg=2.44\n",
            "[118 | 138.47] loss=1.96 avg=2.43\n",
            "[119 | 139.40] loss=1.84 avg=2.42\n",
            "[120 | 140.33] loss=2.49 avg=2.43\n",
            "[121 | 141.26] loss=1.63 avg=2.41\n",
            "[122 | 142.19] loss=2.49 avg=2.42\n",
            "[123 | 143.12] loss=2.08 avg=2.41\n",
            "[124 | 144.05] loss=1.63 avg=2.40\n",
            "[125 | 144.97] loss=2.10 avg=2.40\n",
            "[126 | 145.91] loss=3.43 avg=2.41\n",
            "[127 | 146.84] loss=2.68 avg=2.41\n",
            "[128 | 147.78] loss=1.55 avg=2.40\n",
            "[129 | 148.72] loss=2.65 avg=2.41\n",
            "[130 | 149.66] loss=1.87 avg=2.40\n",
            "[131 | 150.59] loss=3.37 avg=2.41\n",
            "[132 | 151.52] loss=1.48 avg=2.40\n",
            "[133 | 152.44] loss=2.43 avg=2.40\n",
            "[134 | 153.37] loss=3.28 avg=2.41\n",
            "[135 | 154.31] loss=1.49 avg=2.40\n",
            "[136 | 155.24] loss=1.95 avg=2.39\n",
            "[137 | 156.17] loss=2.05 avg=2.39\n",
            "[138 | 157.10] loss=2.05 avg=2.38\n",
            "[139 | 158.04] loss=2.11 avg=2.38\n",
            "[140 | 158.97] loss=1.66 avg=2.37\n",
            "[141 | 159.90] loss=1.82 avg=2.36\n",
            "[142 | 160.82] loss=2.41 avg=2.36\n",
            "[143 | 161.75] loss=1.53 avg=2.35\n",
            "[144 | 162.68] loss=2.33 avg=2.35\n",
            "[145 | 163.62] loss=1.95 avg=2.35\n",
            "[146 | 164.56] loss=1.88 avg=2.34\n",
            "[147 | 165.50] loss=1.91 avg=2.34\n",
            "[148 | 166.43] loss=1.93 avg=2.33\n",
            "[149 | 167.36] loss=1.72 avg=2.32\n",
            "[150 | 168.31] loss=2.56 avg=2.33\n",
            "[151 | 169.25] loss=2.28 avg=2.32\n",
            "[152 | 170.19] loss=1.94 avg=2.32\n",
            "[153 | 171.12] loss=2.83 avg=2.33\n",
            "[154 | 172.05] loss=1.50 avg=2.32\n",
            "[155 | 172.98] loss=1.85 avg=2.31\n",
            "[156 | 173.92] loss=1.74 avg=2.30\n",
            "[157 | 174.84] loss=1.63 avg=2.29\n",
            "[158 | 175.78] loss=1.65 avg=2.29\n",
            "[159 | 176.71] loss=2.11 avg=2.28\n",
            "[160 | 177.64] loss=1.82 avg=2.28\n",
            "[161 | 178.57] loss=1.92 avg=2.27\n",
            "[162 | 179.49] loss=1.85 avg=2.27\n",
            "[163 | 180.41] loss=2.33 avg=2.27\n",
            "[164 | 181.34] loss=1.66 avg=2.26\n",
            "[165 | 182.27] loss=1.46 avg=2.25\n",
            "[166 | 183.19] loss=1.62 avg=2.24\n",
            "[167 | 184.12] loss=1.62 avg=2.24\n",
            "[168 | 185.04] loss=2.43 avg=2.24\n",
            "[169 | 185.97] loss=1.23 avg=2.23\n",
            "[170 | 186.89] loss=1.78 avg=2.22\n",
            "[171 | 187.83] loss=2.42 avg=2.22\n",
            "[172 | 188.76] loss=1.61 avg=2.22\n",
            "[173 | 189.70] loss=1.67 avg=2.21\n",
            "[174 | 190.63] loss=2.53 avg=2.21\n",
            "[175 | 191.56] loss=1.82 avg=2.21\n",
            "[176 | 192.49] loss=1.53 avg=2.20\n",
            "[177 | 193.43] loss=1.21 avg=2.19\n",
            "[178 | 194.35] loss=2.13 avg=2.19\n",
            "[179 | 195.28] loss=2.07 avg=2.19\n",
            "[180 | 196.21] loss=1.44 avg=2.18\n",
            "[181 | 197.14] loss=2.44 avg=2.18\n",
            "[182 | 198.07] loss=2.22 avg=2.18\n",
            "[183 | 199.00] loss=1.86 avg=2.18\n",
            "[184 | 199.94] loss=2.09 avg=2.18\n",
            "[185 | 200.87] loss=2.00 avg=2.17\n",
            "[186 | 201.81] loss=2.12 avg=2.17\n",
            "[187 | 202.74] loss=1.32 avg=2.16\n",
            "[188 | 203.67] loss=2.10 avg=2.16\n",
            "[189 | 204.59] loss=1.52 avg=2.15\n",
            "[190 | 205.52] loss=1.69 avg=2.15\n",
            "[191 | 206.46] loss=2.13 avg=2.15\n",
            "[192 | 207.39] loss=2.10 avg=2.15\n",
            "[193 | 208.32] loss=2.32 avg=2.15\n",
            "[194 | 209.25] loss=2.89 avg=2.16\n",
            "[195 | 210.18] loss=1.48 avg=2.15\n",
            "[196 | 211.12] loss=1.45 avg=2.14\n",
            "[197 | 212.05] loss=2.73 avg=2.15\n",
            "[198 | 212.97] loss=1.91 avg=2.15\n",
            "[199 | 213.90] loss=1.58 avg=2.14\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            "002\n",
            "\n",
            "(1)\n",
            "\n",
            "where\n",
            "\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "and\n",
            "(v) = 0.\n",
            "\n",
            "In this case, we can perform the\n",
            "\n",
            "transition operator\n",
            "\n",
            "where\n",
            "S(x, v, vˆ) =\n",
            "\n",
            "(x)\n",
            "\n",
            "[200 | 235.36] loss=2.05 avg=2.14\n",
            "[201 | 236.29] loss=2.65 avg=2.15\n",
            "[202 | 237.21] loss=1.34 avg=2.14\n",
            "[203 | 238.14] loss=1.23 avg=2.13\n",
            "[204 | 239.06] loss=1.69 avg=2.12\n",
            "[205 | 240.00] loss=1.38 avg=2.11\n",
            "[206 | 240.94] loss=1.44 avg=2.10\n",
            "[207 | 241.87] loss=2.10 avg=2.10\n",
            "[208 | 242.81] loss=1.89 avg=2.10\n",
            "[209 | 243.74] loss=2.23 avg=2.10\n",
            "[210 | 244.67] loss=1.67 avg=2.10\n",
            "[211 | 245.60] loss=1.49 avg=2.09\n",
            "[212 | 246.53] loss=1.76 avg=2.09\n",
            "[213 | 247.46] loss=1.46 avg=2.08\n",
            "[214 | 248.39] loss=1.55 avg=2.07\n",
            "[215 | 249.32] loss=1.73 avg=2.07\n",
            "[216 | 250.25] loss=1.28 avg=2.06\n",
            "[217 | 251.19] loss=2.06 avg=2.06\n",
            "[218 | 252.11] loss=1.38 avg=2.05\n",
            "[219 | 253.04] loss=1.70 avg=2.05\n",
            "[220 | 253.96] loss=1.93 avg=2.05\n",
            "[221 | 254.90] loss=1.72 avg=2.05\n",
            "[222 | 255.83] loss=1.43 avg=2.04\n",
            "[223 | 256.77] loss=1.31 avg=2.03\n",
            "[224 | 257.70] loss=1.37 avg=2.02\n",
            "[225 | 258.64] loss=1.23 avg=2.01\n",
            "[226 | 259.58] loss=1.39 avg=2.01\n",
            "[227 | 260.51] loss=1.38 avg=2.00\n",
            "[228 | 261.45] loss=1.91 avg=2.00\n",
            "[229 | 262.38] loss=1.49 avg=1.99\n",
            "[230 | 263.31] loss=2.82 avg=2.00\n",
            "[231 | 264.25] loss=1.32 avg=1.99\n",
            "[232 | 265.17] loss=1.44 avg=1.99\n",
            "[233 | 266.10] loss=1.64 avg=1.98\n",
            "[234 | 267.03] loss=1.49 avg=1.98\n",
            "[235 | 267.96] loss=1.15 avg=1.97\n",
            "[236 | 268.90] loss=1.86 avg=1.97\n",
            "[237 | 269.83] loss=1.50 avg=1.96\n",
            "[238 | 270.76] loss=1.42 avg=1.96\n",
            "[239 | 271.70] loss=1.60 avg=1.95\n",
            "[240 | 272.61] loss=1.37 avg=1.95\n",
            "[241 | 273.55] loss=1.34 avg=1.94\n",
            "[242 | 274.48] loss=1.20 avg=1.93\n",
            "[243 | 275.40] loss=1.24 avg=1.93\n",
            "[244 | 276.32] loss=1.64 avg=1.92\n",
            "[245 | 277.25] loss=1.20 avg=1.91\n",
            "[246 | 278.18] loss=1.91 avg=1.91\n",
            "[247 | 279.11] loss=1.71 avg=1.91\n",
            "[248 | 280.04] loss=1.19 avg=1.90\n",
            "[249 | 280.96] loss=1.61 avg=1.90\n",
            "[250 | 281.90] loss=1.61 avg=1.90\n",
            "[251 | 282.83] loss=1.36 avg=1.89\n",
            "[252 | 283.76] loss=1.63 avg=1.89\n",
            "[253 | 284.69] loss=2.02 avg=1.89\n",
            "[254 | 285.64] loss=1.33 avg=1.88\n",
            "[255 | 286.58] loss=1.04 avg=1.88\n",
            "[256 | 287.51] loss=1.20 avg=1.87\n",
            "[257 | 288.45] loss=1.43 avg=1.86\n",
            "[258 | 289.38] loss=1.03 avg=1.85\n",
            "[259 | 290.30] loss=1.04 avg=1.85\n",
            "[260 | 291.24] loss=1.84 avg=1.85\n",
            "[261 | 292.17] loss=2.42 avg=1.85\n",
            "[262 | 293.11] loss=1.92 avg=1.85\n",
            "[263 | 294.04] loss=1.78 avg=1.85\n",
            "[264 | 294.97] loss=1.89 avg=1.85\n",
            "[265 | 295.90] loss=1.04 avg=1.84\n",
            "[266 | 296.83] loss=1.08 avg=1.84\n",
            "[267 | 297.76] loss=2.00 avg=1.84\n",
            "[268 | 298.71] loss=1.56 avg=1.83\n",
            "[269 | 299.64] loss=1.78 avg=1.83\n",
            "[270 | 300.58] loss=2.13 avg=1.84\n",
            "[271 | 301.52] loss=1.21 avg=1.83\n",
            "[272 | 302.45] loss=1.03 avg=1.82\n",
            "[273 | 303.39] loss=2.25 avg=1.83\n",
            "[274 | 304.33] loss=1.13 avg=1.82\n",
            "[275 | 305.26] loss=1.66 avg=1.82\n",
            "[276 | 306.18] loss=1.40 avg=1.81\n",
            "[277 | 307.11] loss=1.11 avg=1.80\n",
            "[278 | 308.04] loss=1.41 avg=1.80\n",
            "[279 | 308.98] loss=2.19 avg=1.80\n",
            "[280 | 309.91] loss=1.21 avg=1.80\n",
            "[281 | 310.84] loss=0.96 avg=1.79\n",
            "[282 | 311.77] loss=1.13 avg=1.78\n",
            "[283 | 312.70] loss=0.89 avg=1.77\n",
            "[284 | 313.64] loss=1.72 avg=1.77\n",
            "[285 | 314.57] loss=1.03 avg=1.76\n",
            "[286 | 315.50] loss=2.07 avg=1.77\n",
            "[287 | 316.43] loss=0.93 avg=1.76\n",
            "[288 | 317.36] loss=1.32 avg=1.75\n",
            "[289 | 318.28] loss=0.93 avg=1.75\n",
            "[290 | 319.21] loss=1.28 avg=1.74\n",
            "[291 | 320.14] loss=2.47 avg=1.75\n",
            "[292 | 321.06] loss=1.72 avg=1.75\n",
            "[293 | 321.99] loss=0.88 avg=1.74\n",
            "[294 | 322.92] loss=1.20 avg=1.73\n",
            "[295 | 323.86] loss=0.93 avg=1.72\n",
            "[296 | 324.80] loss=2.07 avg=1.73\n",
            "[297 | 325.74] loss=0.84 avg=1.72\n",
            "[298 | 326.67] loss=1.90 avg=1.72\n",
            "[299 | 327.60] loss=1.64 avg=1.72\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            "S.\n",
            "The first step in the identification of the chemotactic regime is to compare the two regimes. In the first case, we compare the two regimes, while in the second case, we compare the chemotactic regime. In the first case, we compare the two regimes, while in the second case, we compare the chemotactic regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we compare the two regimes, while in the second case, we compare the regime. In the first case, we\n",
            "\n",
            "[300 | 348.94] loss=1.11 avg=1.71\n",
            "[301 | 349.86] loss=1.62 avg=1.71\n",
            "[302 | 350.79] loss=1.19 avg=1.71\n",
            "[303 | 351.72] loss=0.70 avg=1.70\n",
            "[304 | 352.65] loss=1.82 avg=1.70\n",
            "[305 | 353.58] loss=0.90 avg=1.69\n",
            "[306 | 354.50] loss=0.91 avg=1.68\n",
            "[307 | 355.43] loss=1.17 avg=1.68\n",
            "[308 | 356.35] loss=0.75 avg=1.67\n",
            "[309 | 357.28] loss=2.11 avg=1.67\n",
            "[310 | 358.21] loss=0.94 avg=1.66\n",
            "[311 | 359.13] loss=1.06 avg=1.66\n",
            "[312 | 360.06] loss=1.33 avg=1.65\n",
            "[313 | 360.99] loss=1.52 avg=1.65\n",
            "[314 | 361.93] loss=1.02 avg=1.65\n",
            "[315 | 362.86] loss=0.63 avg=1.63\n",
            "[316 | 363.79] loss=1.17 avg=1.63\n",
            "[317 | 364.74] loss=0.87 avg=1.62\n",
            "[318 | 365.68] loss=1.61 avg=1.62\n",
            "[319 | 366.61] loss=1.14 avg=1.62\n",
            "[320 | 367.54] loss=0.88 avg=1.61\n",
            "[321 | 368.47] loss=1.66 avg=1.61\n",
            "[322 | 369.40] loss=0.88 avg=1.60\n",
            "[323 | 370.32] loss=0.77 avg=1.59\n",
            "[324 | 371.26] loss=1.39 avg=1.59\n",
            "[325 | 372.19] loss=1.60 avg=1.59\n",
            "[326 | 373.12] loss=0.89 avg=1.58\n",
            "[327 | 374.05] loss=1.57 avg=1.58\n",
            "[328 | 374.98] loss=1.62 avg=1.58\n",
            "[329 | 375.91] loss=2.22 avg=1.59\n",
            "[330 | 376.84] loss=1.21 avg=1.59\n",
            "[331 | 377.77] loss=1.09 avg=1.58\n",
            "[332 | 378.70] loss=1.68 avg=1.58\n",
            "[333 | 379.64] loss=0.57 avg=1.57\n",
            "[334 | 380.57] loss=0.94 avg=1.57\n",
            "[335 | 381.51] loss=0.59 avg=1.56\n",
            "[336 | 382.44] loss=1.25 avg=1.55\n",
            "[337 | 383.38] loss=1.40 avg=1.55\n",
            "[338 | 384.31] loss=0.87 avg=1.54\n",
            "[339 | 385.24] loss=0.54 avg=1.53\n",
            "[340 | 386.17] loss=1.17 avg=1.53\n",
            "[341 | 387.10] loss=0.98 avg=1.52\n",
            "[342 | 388.04] loss=1.51 avg=1.52\n",
            "[343 | 388.96] loss=0.44 avg=1.51\n",
            "[344 | 389.89] loss=1.37 avg=1.51\n",
            "[345 | 390.81] loss=1.65 avg=1.51\n",
            "[346 | 391.75] loss=1.73 avg=1.51\n",
            "[347 | 392.67] loss=1.36 avg=1.51\n",
            "[348 | 393.61] loss=1.15 avg=1.51\n",
            "[349 | 394.54] loss=0.94 avg=1.50\n",
            "[350 | 395.47] loss=1.27 avg=1.50\n",
            "[351 | 396.39] loss=1.38 avg=1.50\n",
            "[352 | 397.32] loss=0.92 avg=1.49\n",
            "[353 | 398.25] loss=1.13 avg=1.49\n",
            "[354 | 399.17] loss=1.38 avg=1.49\n",
            "[355 | 400.10] loss=0.82 avg=1.48\n",
            "[356 | 401.03] loss=1.94 avg=1.49\n",
            "[357 | 401.95] loss=0.82 avg=1.48\n",
            "[358 | 402.87] loss=0.41 avg=1.47\n",
            "[359 | 403.80] loss=2.16 avg=1.48\n",
            "[360 | 404.73] loss=2.05 avg=1.48\n",
            "[361 | 405.66] loss=0.86 avg=1.48\n",
            "[362 | 406.60] loss=1.46 avg=1.48\n",
            "[363 | 407.53] loss=1.14 avg=1.47\n",
            "[364 | 408.46] loss=1.03 avg=1.47\n",
            "[365 | 409.40] loss=1.86 avg=1.47\n",
            "[366 | 410.33] loss=1.84 avg=1.48\n",
            "[367 | 411.27] loss=1.29 avg=1.47\n",
            "[368 | 412.19] loss=0.92 avg=1.47\n",
            "[369 | 413.12] loss=2.56 avg=1.48\n",
            "[370 | 414.05] loss=0.87 avg=1.47\n",
            "[371 | 414.98] loss=1.09 avg=1.47\n",
            "[372 | 415.90] loss=0.86 avg=1.46\n",
            "[373 | 416.83] loss=1.37 avg=1.46\n",
            "[374 | 417.77] loss=1.08 avg=1.46\n",
            "[375 | 418.70] loss=1.24 avg=1.46\n",
            "[376 | 419.63] loss=1.53 avg=1.46\n",
            "[377 | 420.56] loss=1.00 avg=1.45\n",
            "[378 | 421.50] loss=0.83 avg=1.45\n",
            "[379 | 422.42] loss=1.16 avg=1.44\n",
            "[380 | 423.35] loss=1.40 avg=1.44\n",
            "[381 | 424.27] loss=1.45 avg=1.44\n",
            "[382 | 425.21] loss=1.42 avg=1.44\n",
            "[383 | 426.14] loss=1.52 avg=1.44\n",
            "[384 | 427.07] loss=0.55 avg=1.43\n",
            "[385 | 428.00] loss=0.55 avg=1.42\n",
            "[386 | 428.94] loss=2.22 avg=1.43\n",
            "[387 | 429.88] loss=0.53 avg=1.42\n",
            "[388 | 430.81] loss=0.76 avg=1.42\n",
            "[389 | 431.75] loss=0.72 avg=1.41\n",
            "[390 | 432.68] loss=1.05 avg=1.41\n",
            "[391 | 433.61] loss=0.59 avg=1.40\n",
            "[392 | 434.54] loss=1.73 avg=1.40\n",
            "[393 | 435.47] loss=0.94 avg=1.40\n",
            "[394 | 436.39] loss=1.34 avg=1.40\n",
            "[395 | 437.32] loss=0.58 avg=1.39\n",
            "[396 | 438.26] loss=0.55 avg=1.38\n",
            "[397 | 439.19] loss=1.17 avg=1.38\n",
            "[398 | 440.12] loss=1.58 avg=1.38\n",
            "[399 | 441.04] loss=0.65 avg=1.37\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            " orientation, and the direction of alignment of the fibers.\n",
            "In particular, we shall consider a diffusive model for chemotaxis. In particular, we shall consider a diffusion-advection model. In particular, we shall consider a drift-diffusion model. In particular, we shall consider a non-local sensing model. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a directional cue. In particular, we shall consider a directional cue. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a nonsharing non-local non-sensing kernel.\n",
            "In particular, we shall consider a non-local non-sensing kernel for which the kernel distribution is non-zero. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a non-local non-sensing kernel. In particular, we shall consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing nonsharing kernel.\n",
            "In particular, we shall consider a non-local non-sensing kernel for which the kernel distribution is non-zero. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing nonsharing kernel.\n",
            "In particular, we shall not consider a nonsharing nonsharing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing nonsharing kernel.\n",
            "In particular, we shall not consider a nonsharing nonsharing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing nonsharing kernel. In particular, we shall not consider a nonsharing nonsharing kernel.\n",
            "In particular, we shall not consider a nonsharing nonsharing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing nonsharing kernel.\n",
            "In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing nonsharing kernel.\n",
            "In particular, we shall not consider a non-local non-sensing kernel. In particular, we shall not consider a nonsharing non-local non-sensing kernel. In particular, we shall not consider a nonsharing\n",
            "\n",
            "[400 | 462.48] loss=1.44 avg=1.37\n",
            "[401 | 463.41] loss=0.61 avg=1.36\n",
            "[402 | 464.34] loss=0.57 avg=1.36\n",
            "[403 | 465.27] loss=0.79 avg=1.35\n",
            "[404 | 466.20] loss=0.37 avg=1.34\n",
            "[405 | 467.13] loss=0.87 avg=1.34\n",
            "[406 | 468.06] loss=0.73 avg=1.33\n",
            "[407 | 468.99] loss=1.05 avg=1.33\n",
            "[408 | 469.93] loss=1.21 avg=1.33\n",
            "[409 | 470.86] loss=0.55 avg=1.32\n",
            "[410 | 471.80] loss=2.09 avg=1.33\n",
            "[411 | 472.74] loss=0.61 avg=1.32\n",
            "[412 | 473.66] loss=1.43 avg=1.32\n",
            "[413 | 474.59] loss=0.50 avg=1.31\n",
            "[414 | 475.53] loss=1.01 avg=1.31\n",
            "[415 | 476.46] loss=1.49 avg=1.31\n",
            "[416 | 477.38] loss=0.71 avg=1.30\n",
            "[417 | 478.31] loss=0.53 avg=1.30\n",
            "[418 | 479.24] loss=0.71 avg=1.29\n",
            "[419 | 480.17] loss=0.94 avg=1.29\n",
            "[420 | 481.11] loss=0.92 avg=1.28\n",
            "[421 | 482.03] loss=1.16 avg=1.28\n",
            "[422 | 482.96] loss=0.75 avg=1.28\n",
            "[423 | 483.89] loss=1.20 avg=1.28\n",
            "[424 | 484.82] loss=0.50 avg=1.27\n",
            "[425 | 485.75] loss=0.49 avg=1.26\n",
            "[426 | 486.68] loss=0.81 avg=1.25\n",
            "[427 | 487.60] loss=1.84 avg=1.26\n",
            "[428 | 488.53] loss=0.55 avg=1.25\n",
            "[429 | 489.45] loss=0.38 avg=1.24\n",
            "[430 | 490.38] loss=1.23 avg=1.24\n",
            "[431 | 491.30] loss=0.91 avg=1.24\n",
            "[432 | 492.23] loss=0.77 avg=1.24\n",
            "[433 | 493.15] loss=0.70 avg=1.23\n",
            "[434 | 494.09] loss=1.00 avg=1.23\n",
            "[435 | 495.01] loss=1.86 avg=1.24\n",
            "[436 | 495.95] loss=1.14 avg=1.23\n",
            "[437 | 496.88] loss=0.73 avg=1.23\n",
            "[438 | 497.82] loss=0.51 avg=1.22\n",
            "[439 | 498.76] loss=0.62 avg=1.22\n",
            "[440 | 499.68] loss=1.04 avg=1.21\n",
            "[441 | 500.61] loss=1.39 avg=1.22\n",
            "[442 | 501.54] loss=0.60 avg=1.21\n",
            "[443 | 502.47] loss=0.99 avg=1.21\n",
            "[444 | 503.40] loss=1.10 avg=1.21\n",
            "[445 | 504.33] loss=1.03 avg=1.20\n",
            "[446 | 505.26] loss=0.93 avg=1.20\n",
            "[447 | 506.19] loss=0.89 avg=1.20\n",
            "[448 | 507.12] loss=0.77 avg=1.19\n",
            "[449 | 508.05] loss=1.22 avg=1.19\n",
            "[450 | 508.98] loss=0.78 avg=1.19\n",
            "[451 | 509.90] loss=0.68 avg=1.18\n",
            "[452 | 510.83] loss=0.34 avg=1.18\n",
            "[453 | 511.78] loss=0.82 avg=1.17\n",
            "[454 | 512.71] loss=0.49 avg=1.17\n",
            "[455 | 513.64] loss=0.92 avg=1.16\n",
            "[456 | 514.58] loss=1.10 avg=1.16\n",
            "[457 | 515.53] loss=0.68 avg=1.16\n",
            "[458 | 516.46] loss=0.36 avg=1.15\n",
            "[459 | 517.40] loss=0.37 avg=1.14\n",
            "[460 | 518.34] loss=1.00 avg=1.14\n",
            "[461 | 519.27] loss=0.76 avg=1.14\n",
            "[462 | 520.19] loss=0.32 avg=1.13\n",
            "[463 | 521.12] loss=0.35 avg=1.12\n",
            "[464 | 522.05] loss=0.32 avg=1.11\n",
            "[465 | 522.98] loss=0.93 avg=1.11\n",
            "[466 | 523.91] loss=0.78 avg=1.11\n",
            "[467 | 524.84] loss=0.55 avg=1.10\n",
            "[468 | 525.77] loss=0.70 avg=1.10\n",
            "[469 | 526.70] loss=0.80 avg=1.09\n",
            "[470 | 527.62] loss=0.68 avg=1.09\n",
            "[471 | 528.54] loss=0.68 avg=1.09\n",
            "[472 | 529.46] loss=0.42 avg=1.08\n",
            "[473 | 530.39] loss=0.43 avg=1.07\n",
            "[474 | 531.32] loss=0.63 avg=1.07\n",
            "[475 | 532.24] loss=0.56 avg=1.06\n",
            "[476 | 533.17] loss=0.63 avg=1.06\n",
            "[477 | 534.09] loss=1.12 avg=1.06\n",
            "[478 | 535.01] loss=0.33 avg=1.05\n",
            "[479 | 535.95] loss=0.87 avg=1.05\n",
            "[480 | 536.88] loss=0.43 avg=1.04\n",
            "[481 | 537.82] loss=1.60 avg=1.05\n",
            "[482 | 538.76] loss=0.51 avg=1.04\n",
            "[483 | 539.70] loss=0.76 avg=1.04\n",
            "[484 | 540.63] loss=0.46 avg=1.04\n",
            "[485 | 541.55] loss=0.71 avg=1.03\n",
            "[486 | 542.49] loss=1.91 avg=1.04\n",
            "[487 | 543.42] loss=1.45 avg=1.05\n",
            "[488 | 544.35] loss=0.51 avg=1.04\n",
            "[489 | 545.28] loss=0.66 avg=1.04\n",
            "[490 | 546.20] loss=0.39 avg=1.03\n",
            "[491 | 547.13] loss=0.88 avg=1.03\n",
            "[492 | 548.06] loss=0.40 avg=1.02\n",
            "[493 | 549.00] loss=0.44 avg=1.02\n",
            "[494 | 549.92] loss=0.51 avg=1.01\n",
            "[495 | 550.85] loss=1.18 avg=1.01\n",
            "[496 | 551.77] loss=0.48 avg=1.01\n",
            "[497 | 552.71] loss=0.48 avg=1.00\n",
            "[498 | 553.64] loss=0.94 avg=1.00\n",
            "[499 | 554.57] loss=0.60 avg=1.00\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            ", the average speed of the fibers is given by\n",
            "the momentum\n",
            "T[q, S](x, v, vˆ) = c(x)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ Z\n",
            "R+\n",
            "γq(λ) q(x + λvˆ, vˆ) dλ ψ(v).\n",
            "In particular, the velocity\n",
            "T[q, S] = T0(q, v, vˆ)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) q(x + λvˆ, vˆ) dλ =\n",
            "0\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ\n",
            "(1)\n",
            "and the average\n",
            "T0[q, S] = T0(q, v, vˆ)\n",
            "T0(k, v, vˆ)\n",
            "T0(a) =\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "\n",
            "h\n",
            "\n",
            "| Γ\n",
            "S\n",
            "e\n",
            "i\n",
            "|\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γq\n",
            "k\n",
            "h\n",
            "I\n",
            "(a) vˆ =\n",
            "I(a)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γq\n",
            "k\n",
            "\n",
            "I\n",
            "(b) vˆ =\n",
            "I(b)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "\n",
            "h\n",
            "I\n",
            "(c) vˆ =\n",
            "I(c)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "\n",
            "h\n",
            "I\n",
            "(d) vˆ =\n",
            "I(d)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(e) vˆ =\n",
            "I(e)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(f) vˆ =\n",
            "I(f)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(g) vˆ =\n",
            "I(g)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(h) vˆ =\n",
            "I(h)\n",
            "Z\n",
            "R+\n",
            "γS (λ)S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(i) vˆ =\n",
            "I(i)\n",
            "Z\n",
            "R+\n",
            "γS (λ) S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(k) vˆ =\n",
            "I(k)\n",
            "Z\n",
            "R+\n",
            "γS (λ) S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(l) vˆ =\n",
            "I(l)\n",
            "Z\n",
            "R+\n",
            "γS (λ) S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(m) vˆ =\n",
            "\n",
            "I(m)\n",
            "Z\n",
            "R+\n",
            "γS (λ) S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(n) vˆ =\n",
            "\n",
            "I(n)\n",
            "Z\n",
            "R+\n",
            "γS (λ) S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "q\n",
            "k\n",
            "h\n",
            "I\n",
            "(o) vˆ =\n",
            "\n",
            "I(o)\n",
            "Z\n",
            "R+\n",
            "γS (λ) S(x + λvˆ) dλ i = 1\n",
            "Γ\n",
            "Γ\n",
            "\n",
            "[500 | 576.77] loss=0.66 avg=0.99\n",
            "[501 | 577.70] loss=0.77 avg=0.99\n",
            "[502 | 578.62] loss=0.26 avg=0.98\n",
            "[503 | 579.54] loss=0.15 avg=0.98\n",
            "[504 | 580.47] loss=1.50 avg=0.98\n",
            "[505 | 581.40] loss=0.53 avg=0.98\n",
            "[506 | 582.33] loss=0.56 avg=0.97\n",
            "[507 | 583.26] loss=0.41 avg=0.97\n",
            "[508 | 584.20] loss=0.32 avg=0.96\n",
            "[509 | 585.13] loss=0.35 avg=0.95\n",
            "[510 | 586.06] loss=0.74 avg=0.95\n",
            "[511 | 586.99] loss=0.46 avg=0.95\n",
            "[512 | 587.93] loss=0.78 avg=0.95\n",
            "[513 | 588.86] loss=0.67 avg=0.94\n",
            "[514 | 589.79] loss=0.45 avg=0.94\n",
            "[515 | 590.73] loss=0.97 avg=0.94\n",
            "[516 | 591.66] loss=0.94 avg=0.94\n",
            "[517 | 592.58] loss=0.36 avg=0.93\n",
            "[518 | 593.51] loss=0.69 avg=0.93\n",
            "[519 | 594.44] loss=1.25 avg=0.93\n",
            "[520 | 595.36] loss=0.78 avg=0.93\n",
            "[521 | 596.29] loss=0.58 avg=0.93\n",
            "[522 | 597.23] loss=0.42 avg=0.92\n",
            "[523 | 598.16] loss=0.43 avg=0.92\n",
            "[524 | 599.09] loss=0.30 avg=0.91\n",
            "[525 | 600.03] loss=0.62 avg=0.91\n",
            "[526 | 600.96] loss=0.21 avg=0.90\n",
            "[527 | 601.90] loss=0.57 avg=0.90\n",
            "[528 | 602.83] loss=0.47 avg=0.89\n",
            "[529 | 603.77] loss=0.61 avg=0.89\n",
            "[530 | 604.71] loss=0.69 avg=0.89\n",
            "[531 | 605.64] loss=0.57 avg=0.89\n",
            "[532 | 606.57] loss=0.34 avg=0.88\n",
            "[533 | 607.49] loss=0.95 avg=0.88\n",
            "[534 | 608.42] loss=0.93 avg=0.88\n",
            "[535 | 609.34] loss=1.00 avg=0.88\n",
            "[536 | 610.27] loss=0.45 avg=0.88\n",
            "[537 | 611.20] loss=0.53 avg=0.88\n",
            "[538 | 612.12] loss=0.29 avg=0.87\n",
            "[539 | 613.05] loss=0.27 avg=0.86\n",
            "[540 | 613.97] loss=0.60 avg=0.86\n",
            "[541 | 614.90] loss=0.42 avg=0.86\n",
            "[542 | 615.82] loss=0.22 avg=0.85\n",
            "[543 | 616.74] loss=0.41 avg=0.85\n",
            "[544 | 617.67] loss=0.17 avg=0.84\n",
            "[545 | 618.59] loss=0.43 avg=0.83\n",
            "[546 | 619.52] loss=0.47 avg=0.83\n",
            "[547 | 620.44] loss=0.69 avg=0.83\n",
            "[548 | 621.36] loss=0.27 avg=0.82\n",
            "[549 | 622.29] loss=0.65 avg=0.82\n",
            "[550 | 623.22] loss=1.12 avg=0.83\n",
            "[551 | 624.15] loss=0.54 avg=0.82\n",
            "[552 | 625.08] loss=0.46 avg=0.82\n",
            "[553 | 626.02] loss=0.62 avg=0.82\n",
            "[554 | 626.96] loss=0.27 avg=0.81\n",
            "[555 | 627.89] loss=0.35 avg=0.81\n",
            "[556 | 628.82] loss=0.25 avg=0.80\n",
            "[557 | 629.76] loss=0.41 avg=0.80\n",
            "[558 | 630.69] loss=0.26 avg=0.79\n",
            "[559 | 631.62] loss=0.68 avg=0.79\n",
            "[560 | 632.56] loss=0.24 avg=0.78\n",
            "[561 | 633.49] loss=0.21 avg=0.78\n",
            "[562 | 634.42] loss=0.30 avg=0.77\n",
            "[563 | 635.35] loss=0.32 avg=0.77\n",
            "[564 | 636.29] loss=0.31 avg=0.77\n",
            "[565 | 637.21] loss=0.36 avg=0.76\n",
            "[566 | 638.14] loss=0.52 avg=0.76\n",
            "[567 | 639.08] loss=0.16 avg=0.75\n",
            "[568 | 640.01] loss=0.50 avg=0.75\n",
            "[569 | 640.95] loss=0.73 avg=0.75\n",
            "[570 | 641.88] loss=0.50 avg=0.75\n",
            "[571 | 642.81] loss=0.43 avg=0.74\n",
            "[572 | 643.75] loss=0.68 avg=0.74\n",
            "[573 | 644.67] loss=0.61 avg=0.74\n",
            "[574 | 645.61] loss=0.13 avg=0.74\n",
            "[575 | 646.54] loss=0.21 avg=0.73\n",
            "[576 | 647.47] loss=0.34 avg=0.73\n",
            "[577 | 648.39] loss=0.33 avg=0.72\n",
            "[578 | 649.32] loss=0.22 avg=0.72\n",
            "[579 | 650.26] loss=0.52 avg=0.72\n",
            "[580 | 651.19] loss=0.26 avg=0.71\n",
            "[581 | 652.12] loss=0.51 avg=0.71\n",
            "[582 | 653.05] loss=0.76 avg=0.71\n",
            "[583 | 653.97] loss=0.73 avg=0.71\n",
            "[584 | 654.91] loss=0.34 avg=0.71\n",
            "[585 | 655.84] loss=0.42 avg=0.70\n",
            "[586 | 656.76] loss=0.48 avg=0.70\n",
            "[587 | 657.69] loss=0.34 avg=0.70\n",
            "[588 | 658.62] loss=0.33 avg=0.69\n",
            "[589 | 659.54] loss=0.56 avg=0.69\n",
            "[590 | 660.47] loss=0.52 avg=0.69\n",
            "[591 | 661.40] loss=0.32 avg=0.69\n",
            "[592 | 662.32] loss=0.19 avg=0.68\n",
            "[593 | 663.25] loss=0.21 avg=0.68\n",
            "[594 | 664.16] loss=0.16 avg=0.67\n",
            "[595 | 665.10] loss=0.69 avg=0.67\n",
            "[596 | 666.02] loss=0.27 avg=0.67\n",
            "[597 | 666.95] loss=0.17 avg=0.66\n",
            "[598 | 667.88] loss=0.93 avg=0.67\n",
            "[599 | 668.82] loss=0.24 avg=0.66\n",
            "Generating samples...\n",
            "======== SAMPLE 1 ========\n",
            " values, and the kernel\n",
            "(x, v, vˆ) is the average of the variance of the\n",
            "choosing direction. The mean direction is given by the mean/(vˆ)\n",
            "distribution\n",
            "0, v, vˆ =\n",
            "20\n",
            "4\n",
            "4\n",
            "(1) (2) (3)\n",
            "(4)\n",
            "(5)\n",
            "and the variance, i.e., the\n",
            "turning kernel, given by the\n",
            "turning(k(x)) =\n",
            "90\n",
            "Γ\n",
            "0\n",
            "q(x, v, vˆ)\n",
            "0\n",
            "that is the turning operator, is given by\n",
            "UT\n",
            "dv = V\n",
            "0\n",
            "Γ\n",
            "q(x, v, vˆ)\n",
            "1\n",
            "that is the variance of the choice of the\n",
            "turning operator, given by UT\n",
            "dvˆ = UT\n",
            "2\n",
            "Γ\n",
            "q(x, v, vˆ)\n",
            "2\n",
            "that is the turning\n",
            "direction given by UT\n",
            "0\n",
            "Γ\n",
            "q\n",
            "(x, v, vˆ)\n",
            "(6) (7)\n",
            "and the mean velocity\n",
            "Vˆ\n",
            "0\n",
            "(max) =\n",
            "70\n",
            "Γ\n",
            "q\n",
            "(x, v, vˆ)\n",
            "(8)\n",
            "and the\n",
            "U1\n",
            "q\n",
            "(x, v, vˆ) =\n",
            "70\n",
            "Γ\n",
            "q\n",
            "(x, v, vˆ)\n",
            "(9)\n",
            "and the\n",
            "D\n",
            "0\n",
            "T\n",
            "(ξ) =\n",
            "U¯\n",
            "(ξ)\n",
            "∇ · Dq\n",
            "0\n",
            "T\n",
            "(ξ)\n",
            "· (v0, v1) + ξv (ξ) (v0, v1).\n",
            "Re-scaling the space variable as in (6), we have\n",
            "D\n",
            "0\n",
            "T\n",
            "(ξ) =\n",
            "U¯\n",
            "ξ\n",
            "Γ\n",
            "q\n",
            "(ξ)\n",
            "∇ · Dq\n",
            "0\n",
            "T\n",
            "(ξ)\n",
            "· (v0, v1) + ξv (ξ) (v0, v1).\n",
            "The mean direction is given by the variance of the\n",
            "turning direction given by\n",
            "UT (ξ) = UT\n",
            "Γ\n",
            "q\n",
            "(ξ)U¯\n",
            "Γ\n",
            "q\n",
            "(ξ)∇ · Dq\n",
            "0\n",
            "T\n",
            "(ξ)\n",
            ". (10)\n",
            "As a consequence, the macroscopic behavior is strongly affected by the\n",
            "turning operator, that is\n",
            "D\n",
            "0\n",
            "T\n",
            "(ξ) =\n",
            "U¯\n",
            "ξ\n",
            "Γ\n",
            "q\n",
            "(ξ)∇ · Dq\n",
            "0\n",
            "T\n",
            "(ξ)\n",
            ". (11)\n",
            "In particular, the sensing radius of the cells is given by\n",
            "S\n",
            "c(x, y) =\n",
            "Γ\n",
            "q\n",
            "(x, y)U¯\n",
            "Γ\n",
            "q\n",
            "(x, y)Γ\n",
            "both\n",
            "i\n",
            " and ii\n",
            ", v\n",
            ", dv\n",
            ",\n",
            "dvˆ\n",
            ",\n",
            "Γ\n",
            "q\n",
            ",\n",
            "are given by (12) and (13). The chemoattractant has a Gaussian\n",
            "c = c(x, y)\n",
            "and on the left the two values of c both have to be in the\n",
            "same direction. Therefore, the sensing radius of the cells is given by\n",
            "the momentum\n",
            "T = S\n",
            "c(x, y) =\n",
            "v\n",
            "(x, y)\n",
            "Γ\n",
            "q\n",
            "(x, y)\n",
            "vˆ + Γ ii\n",
            "(x, y)\n",
            "iiˆ + Γ\n",
            "q\n",
            "(x, y)\n",
            "iiˆ + Γ\n",
            "v\n",
            "(x, y)\n",
            "(14)\n",
            "and the sensing function Γ =\n",
            "Γ\n",
            "q\n",
            "(x, y)U¯\n",
            "Γ\n",
            ". (15)\n",
            "In particular, when the two sensing functions are independent,\n",
            "when Γ is equal to Γvˆ, we have that the weighted average\n",
            "for the two velocities is given by the momentum\n",
            "T = S\n",
            "c(x, y) =\n",
            "\n",
            "v\n",
            "(x, y)\n",
            "Γ\n",
            "q\n",
            "(x, y)\n",
            "Γ\n",
            "\n",
            "i\n",
            "and\n",
            "k\n",
            ":= vˆ k(x, y)\n",
            ". (16)\n",
            "This translates into\n",
            "k(x, y) =\n",
            "vˆ(x)\n",
            ",\n",
            "that is the kurtosis\n",
            "T\n",
            "(ξ) = vˆ(x)\n",
            ",\n",
            "that is the tach statistic\n",
            "DT\n",
            "(ξ) = u\n",
            "T\n",
            "(ξ)\n",
            "∇T\n",
            "(ξ)\n",
            ". (\n",
            "\n",
            "[600 | 690.44] loss=0.29 avg=0.66\n",
            "[601 | 691.37] loss=0.45 avg=0.66\n",
            "[602 | 692.29] loss=0.27 avg=0.65\n",
            "[603 | 693.22] loss=0.38 avg=0.65\n",
            "[604 | 694.15] loss=0.33 avg=0.65\n",
            "[605 | 695.09] loss=0.89 avg=0.65\n",
            "[606 | 696.01] loss=0.59 avg=0.65\n",
            "[607 | 696.94] loss=0.27 avg=0.64\n",
            "[608 | 697.86] loss=0.61 avg=0.64\n",
            "[609 | 698.80] loss=0.33 avg=0.64\n",
            "[610 | 699.72] loss=0.80 avg=0.64\n",
            "[611 | 700.65] loss=0.49 avg=0.64\n",
            "[612 | 701.58] loss=0.35 avg=0.64\n",
            "[613 | 702.51] loss=0.26 avg=0.63\n",
            "[614 | 703.43] loss=0.47 avg=0.63\n",
            "[615 | 704.36] loss=0.41 avg=0.63\n",
            "[616 | 705.28] loss=0.42 avg=0.63\n",
            "[617 | 706.20] loss=0.69 avg=0.63\n",
            "[618 | 707.13] loss=0.41 avg=0.63\n",
            "[619 | 708.06] loss=0.44 avg=0.62\n",
            "[620 | 709.00] loss=0.23 avg=0.62\n",
            "[621 | 709.94] loss=0.46 avg=0.62\n",
            "[622 | 710.88] loss=0.34 avg=0.62\n",
            "[623 | 711.82] loss=0.34 avg=0.61\n",
            "[624 | 712.74] loss=0.26 avg=0.61\n",
            "[625 | 713.67] loss=0.23 avg=0.61\n",
            "[626 | 714.59] loss=0.35 avg=0.60\n",
            "[627 | 715.52] loss=0.50 avg=0.60\n",
            "[628 | 716.45] loss=0.31 avg=0.60\n",
            "[629 | 717.38] loss=0.36 avg=0.60\n",
            "[630 | 718.31] loss=0.13 avg=0.59\n",
            "[631 | 719.24] loss=0.29 avg=0.59\n",
            "[632 | 720.17] loss=0.20 avg=0.59\n",
            "[633 | 721.10] loss=0.23 avg=0.58\n",
            "[634 | 722.03] loss=0.16 avg=0.58\n",
            "[635 | 722.96] loss=0.14 avg=0.57\n",
            "[636 | 723.90] loss=0.31 avg=0.57\n",
            "[637 | 724.83] loss=0.80 avg=0.57\n",
            "[638 | 725.78] loss=0.31 avg=0.57\n",
            "[639 | 726.72] loss=0.38 avg=0.57\n",
            "[640 | 727.65] loss=0.23 avg=0.57\n",
            "[641 | 728.59] loss=0.24 avg=0.56\n",
            "[642 | 729.53] loss=0.35 avg=0.56\n",
            "[643 | 730.46] loss=0.19 avg=0.56\n",
            "[644 | 731.39] loss=0.16 avg=0.55\n",
            "[645 | 732.32] loss=0.24 avg=0.55\n",
            "[646 | 733.25] loss=0.10 avg=0.54\n",
            "[647 | 734.18] loss=0.40 avg=0.54\n",
            "[648 | 735.11] loss=0.28 avg=0.54\n",
            "[649 | 736.04] loss=0.36 avg=0.54\n",
            "[650 | 736.97] loss=0.27 avg=0.54\n",
            "[651 | 737.90] loss=0.78 avg=0.54\n",
            "interrupted\n",
            "Saving checkpoint/run1/model-652\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z-zAFd2hLQ2V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Step 10: Creating a Training Model directory\n",
        "#Creating a Training Model directory named 'tgmodel'\n",
        "import os\n",
        "run_dir = '/content/gpt-2/models/tgmodel'\n",
        "if not os.path.exists(run_dir):\n",
        "  os.makedirs(run_dir)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-POx-g1Ql76C",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "4a14528a-cb8e-4b8c-9b8a-9a5cfab4c6fe"
      },
      "source": [
        "#@title Step 10A: Copying training Files\n",
        "!cp /content/gpt-2/src/checkpoint/run1/model-1000.data-00000-of-00001 /content/gpt-2/models/tgmodel\n",
        "!cp /content/gpt-2/src/checkpoint/run1/checkpoint /content/gpt-2/models/tgmodel\n",
        "!cp /content/gpt-2/src/checkpoint/run1/model-1000.index /content/gpt-2/models/tgmodel\n",
        "!cp /content/gpt-2/src/checkpoint/run1/model-1000.meta /content/gpt-2/models/tgmodel"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cp: cannot stat '/content/gpt-2/src/checkpoint/run1/model-1000.data-00000-of-00001': No such file or directory\n",
            "cp: cannot stat '/content/gpt-2/src/checkpoint/run1/model-1000.index': No such file or directory\n",
            "cp: cannot stat '/content/gpt-2/src/checkpoint/run1/model-1000.meta': No such file or directory\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hdE9nNH8m7VD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Step 10B: Copying the OpenAI GPT-2 117M Model files\n",
        "!cp /content/gpt-2/models/117M/encoder.json /content/gpt-2/models/tgmodel\n",
        "!cp /content/gpt-2/models/117M/hparams.json /content/gpt-2/models/tgmodel\n",
        "!cp /content/gpt-2/models/117M/vocab.bpe /content/gpt-2/models/tgmodel"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3G8NOUXjMq4u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Step 10C: Renaming the model directories\n",
        "import os\n",
        "!mv /content/gpt-2/models/117M  /content/gpt-2/models/117M_OpenAI\n",
        "!mv /content/gpt-2/models/tgmodel  /content/gpt-2/models/117M"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h3uexz_e4d18",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Step 11: Generating Unconditional Samples\n",
        "import os # import after runtime is restarted\n",
        "os.chdir(\"/content/gpt-2/src\")\n",
        "!python generate_unconditional_samples.py --model_name '117M'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6HI7DuBK4iSU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "aced43aa-7fcc-4bac-99c7-8bfd7ded0028"
      },
      "source": [
        "#@title Step 12: Interactive Context and Completion Examples\n",
        "import os # import after runtime is restarted\n",
        "os.chdir(\"/content/gpt-2/src\")\n",
        "!python interactive_conditional_samples.py --temperature 0.8 --top_k 40 --model_name '117M' --length 50"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From interactive_conditional_samples.py:57: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
            "\n",
            "2020-06-29 09:30:02.273624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
            "2020-06-29 09:30:02.292947: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.293714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2020-06-29 09:30:02.294023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-06-29 09:30:02.295631: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2020-06-29 09:30:02.297301: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2020-06-29 09:30:02.297699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2020-06-29 09:30:02.299362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2020-06-29 09:30:02.300174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2020-06-29 09:30:02.303450: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2020-06-29 09:30:02.303619: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.304415: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.305120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2020-06-29 09:30:02.310474: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299995000 Hz\n",
            "2020-06-29 09:30:02.310737: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1426d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
            "2020-06-29 09:30:02.310775: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
            "2020-06-29 09:30:02.360414: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.361376: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1426f40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
            "2020-06-29 09:30:02.361416: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
            "2020-06-29 09:30:02.361699: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.362523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2020-06-29 09:30:02.362622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-06-29 09:30:02.362681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "2020-06-29 09:30:02.362735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
            "2020-06-29 09:30:02.362790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
            "2020-06-29 09:30:02.362854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
            "2020-06-29 09:30:02.362922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
            "2020-06-29 09:30:02.362980: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
            "2020-06-29 09:30:02.363153: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.364047: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.364759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
            "2020-06-29 09:30:02.364834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-06-29 09:30:02.366467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2020-06-29 09:30:02.366509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 \n",
            "2020-06-29 09:30:02.366530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N \n",
            "2020-06-29 09:30:02.366754: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.367607: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2020-06-29 09:30:02.368323: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2020-06-29 09:30:02.368380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10805 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "WARNING:tensorflow:From interactive_conditional_samples.py:58: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "WARNING:tensorflow:From interactive_conditional_samples.py:60: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:51: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:148: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:152: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/model.py:36: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.\n",
            "\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:64: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:16: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:67: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.random.categorical` instead.\n",
            "WARNING:tensorflow:From interactive_conditional_samples.py:68: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n",
            "\n",
            "Model prompt >>> During such processes, cells sense the environment and respond to external factors that induce a certain direction of motion towards specific targets (taxis): this results in a persistent migration in a certain preferential direction. The guidance cues leading to directed migration may be biochemical or biophysical. Biochemical cues can be, for example, soluble factors or growth factors that give rise to chemotaxis, which involves a mono-directional stimulus. Other cues generating mono-directional stimuli include, for instance, bound ligands to the substratum that induce haptotaxis, durotaxis, that involves migration towards regions with an increasing stiffness of the ECM, electrotaxis, also known as galvanotaxis, that prescribes a directed motion guided by an electric field or current, or phototaxis, referring to the movement oriented by a stimulus of light [34]. Important biophysical cues are some of the properties of the extracellular matrix (ECM), first among all the alignment of collagen fibers and its stiffness. In particular, the fiber alignment is shown to stimulate contact guidance [22, 21]. TL;DR:\n",
            "2020-06-29 09:31:30.405327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
            "======================================== SAMPLE 1 ========================================\n",
            " the ECM of a single tissue is the ECM that is the most effective.\n",
            "\n",
            "To address this concern, we developed a novel imaging and immunostaining scheme that, when activated, induces the conversion of a protein to its exogenous target\n",
            "================================================================================\n",
            "Model prompt >>> Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3.6/contextlib.py\", line 99, in __exit__\n",
            "    self.gen.throw(type, value, traceback)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py\", line 5480, in get_controller\n",
            "    yield g\n",
            "  File \"interactive_conditional_samples.py\", line 73, in interact_model\n",
            "    raw_text = input(\"Model prompt >>> \")\n",
            "KeyboardInterrupt\n",
            "\n",
            "During handling of the above exception, another exception occurred:\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"interactive_conditional_samples.py\", line 91, in <module>\n",
            "    fire.Fire(interact_model)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/fire/core.py\", line 138, in Fire\n",
            "    component_trace = _Fire(component, args, parsed_flag_args, context, name)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/fire/core.py\", line 468, in _Fire\n",
            "    target=component.__name__)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/fire/core.py\", line 672, in _CallAndUpdateTrace\n",
            "    component = fn(*varargs, **kwargs)\n",
            "  File \"interactive_conditional_samples.py\", line 88, in interact_model\n",
            "    print(\"=\" * 80)\n",
            "  File \"/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py\", line 1633, in __exit__\n",
            "    close_thread.start()\n",
            "  File \"/usr/lib/python3.6/threading.py\", line 851, in start\n",
            "    self._started.wait()\n",
            "  File \"/usr/lib/python3.6/threading.py\", line 551, in wait\n",
            "    signaled = self._cond.wait(timeout)\n",
            "  File \"/usr/lib/python3.6/threading.py\", line 295, in wait\n",
            "    waiter.acquire()\n",
            "KeyboardInterrupt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ihVnmXFYB-E7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "bbe7802e-2e06-4c70-debb-bab34bfb0c2e"
      },
      "source": [
        "#@title Additional Tools: Controlling Tokenized Data\n",
        "#Unzip out.npz\n",
        "import zipfile\n",
        "with zipfile.ZipFile('/content/gpt-2/src/out.npz', 'r') as zip_ref:\n",
        "    zip_ref.extractall('/content/gpt-2/src/')\n",
        "\n",
        "#Load arr_0.npy which contains encoded dset\n",
        "import numpy as np\n",
        "f=np.load('/content/gpt-2/src/arr_0.npy')\n",
        "print(f)\n",
        "print(f.shape)\n",
        "for i in range(0,10):\n",
        "    print(f[i])\n",
        "     \n",
        "#We first import encoder.json\n",
        "import json\n",
        "i=0\n",
        "with open(\"/content/gpt-2/models/117M/encoder.json\", \"r\") as read_file:\n",
        "    print(\"Converting the JSON encoded data into a Python dictionary\")\n",
        "    developer = json.load(read_file) #converts the encoded data into a Python dictionary\n",
        "    for key, value in developer.items(): #we parse the decoded json data\n",
        "        i+=1\n",
        "        if(i>10):\n",
        "            break;\n",
        "        print(key, \":\", value)\n",
        "\n",
        "#We will now search for the key and value for each encoded token\n",
        "    for i in range(0,500):\n",
        "        for key, value in developer.items():\n",
        "            if f[i]==value:\n",
        "                print(key, \":\", value)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1212 5644  326 ...   13  198 2682]\n",
            "(29379,)\n",
            "1212\n",
            "5644\n",
            "326\n",
            "11\n",
            "355\n",
            "716\n",
            "78\n",
            "1765\n",
            "1868\n",
            "4778\n",
            "Converting JSON encoded data into Python dictionary\n",
            "! : 0\n",
            "\" : 1\n",
            "# : 2\n",
            "$ : 3\n",
            "% : 4\n",
            "& : 5\n",
            "' : 6\n",
            "( : 7\n",
            ") : 8\n",
            "* : 9\n",
            "This : 1212\n",
            "Ġsuggests : 5644\n",
            "Ġthat : 326\n",
            ", : 11\n",
            "Ġas : 355\n",
            "Ġam : 716\n",
            "o : 78\n",
            "eb : 1765\n",
            "oid : 1868\n",
            "Ġcells : 4778\n",
            "Ġare : 389\n",
            "Ġless : 1342\n",
            "Ġcontract : 2775\n",
            "ile : 576\n",
            ", : 11\n",
            "Ġwhile : 981\n",
            "Ġmes : 18842\n",
            "ench : 24421\n",
            "ym : 4948\n",
            "al : 282\n",
            "Ċ : 198\n",
            "cells : 46342\n",
            "Ġare : 389\n",
            "Ġmore : 517\n",
            "Ġcontract : 2775\n",
            "ile : 576\n",
            ", : 11\n",
            "Ġand : 290\n",
            "Ġthere : 612\n",
            "Ġmay : 743\n",
            "Ġbe : 307\n",
            "Ġa : 257\n",
            "Ġswitching : 15430\n",
            "Ġbetween : 1022\n",
            "Ġam : 716\n",
            "o : 78\n",
            "eb : 1765\n",
            "oid : 1868\n",
            "Ġand : 290\n",
            "Ġmes : 18842\n",
            "ench : 24421\n",
            "ym : 4948\n",
            "al : 282\n",
            "Ċ : 198\n",
            "m : 76\n",
            "igration : 4254\n",
            ", : 11\n",
            "Ġperhaps : 3737\n",
            "Ġthere : 612\n",
            "Ġcan : 460\n",
            "Ġalso : 635\n",
            "Ġbe : 307\n",
            "Ġa : 257\n",
            "Ġswitching : 15430\n",
            "Ġbetween : 1022\n",
            "Ġthe : 262\n",
            "Ġdominance : 18648\n",
            "Ġof : 286\n",
            "Ġchem : 4607\n",
            "ot : 313\n",
            "axis : 22704\n",
            "Ġ( : 357\n",
            "amo : 18811\n",
            "eb : 1765\n",
            "oid : 1868\n",
            "Ċ : 198\n",
            "m : 76\n",
            "igration : 4254\n",
            ") : 8\n",
            "Ġand : 290\n",
            "Ġcontact : 2800\n",
            "Ġguidance : 11154\n",
            "Ġ( : 357\n",
            "mes : 6880\n",
            "ench : 24421\n",
            "ym : 4948\n",
            "al : 282\n",
            "Ġmigration : 13472\n",
            ") : 8\n",
            "Ġ[ : 685\n",
            "60 : 1899\n",
            "]. : 4083\n",
            "ĠOne : 1881\n",
            "Ġof : 286\n",
            "Ġthe : 262\n",
            "Ġmost : 749\n",
            "Ġinteresting : 3499\n",
            "Ġ2 : 362\n",
            "D : 35\n",
            "Ċ : 198\n",
            "platform : 24254\n",
            "s : 82\n",
            ", : 11\n",
            "Ġallowing : 5086\n",
            "Ġto : 284\n",
            "Ġstudy : 2050\n",
            "Ġcontact : 2800\n",
            "Ġguidance : 11154\n",
            "Ġand : 290\n",
            "Ġchem : 4607\n",
            "ot : 313\n",
            "axis : 22704\n",
            ", : 11\n",
            "Ġwas : 373\n",
            "Ġproposed : 5150\n",
            "Ġin : 287\n",
            "Ġ[ : 685\n",
            "57 : 3553\n",
            "], : 4357\n",
            "Ġin : 287\n",
            "Ġwhich : 543\n",
            "Ġthe : 262\n",
            "Ċ : 198\n",
            "authors : 41617\n",
            "Ġdemonstrated : 9555\n",
            "Ġan : 281\n",
            "Ġadditive : 38298\n",
            "Ġeffect : 1245\n",
            "Ġof : 286\n",
            "Ġchemical : 5931\n",
            "Ġgrad : 3915\n",
            "ients : 2334\n",
            "Ġand : 290\n",
            "Ġfiber : 13608\n",
            "Ġalignment : 19114\n",
            "Ġby : 416\n",
            "Ġmeasuring : 15964\n",
            "Ċ : 198\n",
            "the : 1169\n",
            "Ġpersistence : 30802\n",
            "Ġtime : 640\n",
            "; : 26\n",
            "Ġthey : 484\n",
            "Ġalso : 635\n",
            "Ġobserved : 6515\n",
            "Ġthat : 326\n",
            "Ġcells : 4778\n",
            "Ġwere : 547\n",
            "Ġdirected : 7924\n",
            "Ġby : 416\n",
            "Ġfiber : 13608\n",
            "Ġalignment : 19114\n",
            "Ġand : 290\n",
            "Ġthere : 612\n",
            "Ġwas : 373\n",
            "Ċ : 198\n",
            "no : 3919\n",
            "Ġeffect : 1245\n",
            "Ġof : 286\n",
            "Ġthe : 262\n",
            "Ġchemical : 5931\n",
            "Ġgradient : 31312\n",
            "Ġwhen : 618\n",
            "Ġfibers : 26742\n",
            "Ġwere : 547\n",
            "Ġaligned : 19874\n",
            "Ġperpendicular : 47190\n",
            "Ġto : 284\n",
            "Ġit : 340\n",
            ". : 13\n",
            "ĠA : 317\n",
            "Ġsimilar : 2092\n",
            "Ġsetting : 4634\n",
            "Ċ : 198\n",
            "was : 9776\n",
            "Ġalso : 635\n",
            "Ġused : 973\n",
            "Ġfor : 329\n",
            "Ġstudying : 11065\n",
            "Ġthe : 262\n",
            "Ġdependence : 21403\n",
            "Ġof : 286\n",
            "Ġcontact : 2800\n",
            "Ġguidance : 11154\n",
            "Ġon : 319\n",
            "Ġthe : 262\n",
            "Ġcell : 2685\n",
            "Ġcycle : 6772\n",
            "Ġ[ : 685\n",
            "48 : 2780\n",
            "]. : 4083\n",
            "ĠHowever : 2102\n",
            ", : 11\n",
            "ĠIn : 554\n",
            "Ċ : 198\n",
            "the : 1169\n",
            "Ġcase : 1339\n",
            "Ġof : 286\n",
            "Ġdifferent : 1180\n",
            "Ġmulti : 5021\n",
            "- : 12\n",
            "direction : 37295\n",
            "al : 282\n",
            "Ġcues : 25288\n",
            ", : 11\n",
            "Ġtotally : 6635\n",
            "Ġdifferent : 1180\n",
            "Ġscenarios : 13858\n",
            "Ġmay : 743\n",
            "Ġhappen : 1645\n",
            ", : 11\n",
            "Ġe : 304\n",
            ". : 13\n",
            "g : 70\n",
            ". : 13\n",
            "Ġin : 287\n",
            "Ġ[ : 685\n",
            "51 : 4349\n",
            "] : 60\n",
            "Ġit : 340\n",
            "Ġis : 318\n",
            "Ċ : 198\n",
            "shown : 42579\n",
            "Ġthat : 326\n",
            "Ġfor : 329\n",
            "Ġcontact : 2800\n",
            "Ġguidance : 11154\n",
            "Ġand : 290\n",
            "Ġelect : 1742\n",
            "rot : 10599\n",
            "axis : 22704\n",
            "Ġin : 287\n",
            "Ġthe : 262\n",
            "Ġcor : 1162\n",
            "nea : 39718\n",
            ", : 11\n",
            "Ġelect : 1742\n",
            "rot : 10599\n",
            "axis : 22704\n",
            "Ġwins : 7864\n",
            "Ġwhen : 618\n",
            "Ġcompeting : 11780\n",
            "Ċ : 198\n",
            "with : 4480\n",
            "Ġthe : 262\n",
            "Ġdirection : 4571\n",
            "Ġof : 286\n",
            "Ġalignment : 19114\n",
            "Ġof : 286\n",
            "Ġthe : 262\n",
            "Ġfibers : 26742\n",
            ". : 13\n",
            "Ċ : 198\n",
            "Multi : 29800\n",
            "- : 12\n",
            "cue : 15509\n",
            "Ġkinetic : 37892\n",
            "Ġmodel : 2746\n",
            "Ġwith : 351\n",
            "Ġnon : 1729\n",
            "- : 12\n",
            "local : 12001\n",
            "Ġsensing : 34244\n",
            "Ġfor : 329\n",
            "Ġcell : 2685\n",
            "Ċ : 198\n",
            "m : 76\n",
            "igration : 4254\n",
            "Ġon : 319\n",
            "Ġa : 257\n",
            "Ġfibers : 26742\n",
            "Ġnetwork : 3127\n",
            "Ġwith : 351\n",
            "Ġchem : 4607\n",
            "ot : 313\n",
            "axis : 22704\n",
            "Ċ : 198\n",
            "Mart : 13143\n",
            "ina : 1437\n",
            "ĠCon : 1482\n",
            "te : 660\n",
            "ĠâĪ : 18872\n",
            "Ĺ : 245\n",
            "ĠNad : 21877\n",
            "ia : 544\n",
            "ĠL : 406\n",
            "oy : 726\n",
            "ĠâĢ : 564\n",
            "ł : 254\n",
            "âĢ : 447\n",
            "¡ : 94\n",
            "Ċ : 198\n",
            "June : 15749\n",
            "Ġ18 : 1248\n",
            ", : 11\n",
            "Ġ2020 : 12131\n",
            "Ċ : 198\n",
            "Abstract : 23839\n",
            "Ċ : 198\n",
            "C : 34\n",
            "ells : 19187\n",
            "Ġperform : 1620\n",
            "Ġdirected : 7924\n",
            "Ġmotion : 6268\n",
            "Ġin : 287\n",
            "Ġresponse : 2882\n",
            "Ġto : 284\n",
            "Ġexternal : 7097\n",
            "Ġstimuli : 25973\n",
            "Ġthat : 326\n",
            "Ġthey : 484\n",
            "Ġdetect : 4886\n",
            "Ġby : 416\n",
            "Ġsensing : 34244\n",
            "Ċ : 198\n",
            "the : 1169\n",
            "Ġenvironment : 2858\n",
            "Ġwith : 351\n",
            "Ġtheir : 511\n",
            "Ġmembrane : 25019\n",
            "Ġprot : 1237\n",
            "rus : 14932\n",
            "ions : 507\n",
            ". : 13\n",
            "ĠIn : 554\n",
            "Ġparticular : 1948\n",
            ", : 11\n",
            "Ġseveral : 1811\n",
            "Ġbiochemical : 47685\n",
            "Ġand : 290\n",
            "Ġbi : 3182\n",
            "ophysical : 41789\n",
            "Ġcues : 25288\n",
            "Ġgive : 1577\n",
            "Ġrise : 4485\n",
            "Ġto : 284\n",
            "Ġtactic : 18543\n",
            "Ġmigration : 13472\n",
            "Ġin : 287\n",
            "Ġthe : 262\n",
            "Ġdirection : 4571\n",
            "Ġof : 286\n",
            "Ġtheir : 511\n",
            "Ġspecific : 2176\n",
            "Ġtargets : 6670\n",
            ". : 13\n",
            "ĠThis : 770\n",
            "Ġdefines : 15738\n",
            "Ċ : 198\n",
            "a : 64\n",
            "Ġmulti : 5021\n",
            "- : 12\n",
            "cue : 15509\n",
            "Ġenvironment : 2858\n",
            "Ġin : 287\n",
            "Ġwhich : 543\n",
            "Ġcells : 4778\n",
            "Ġhave : 423\n",
            "Ġto : 284\n",
            "Ġsort : 3297\n",
            "Ġand : 290\n",
            "Ġcombine : 12082\n",
            "Ġdifferent : 1180\n",
            ", : 11\n",
            "Ġand : 290\n",
            "Ġpotentially : 6196\n",
            "Ċ : 198\n",
            "competitive : 46131\n",
            ", : 11\n",
            "Ġstimuli : 25973\n",
            ". : 13\n",
            "ĠWe : 775\n",
            "Ġpropose : 18077\n",
            "Ġa : 257\n",
            "Ġnon : 1729\n",
            "- : 12\n",
            "local : 12001\n",
            "Ġkinetic : 37892\n",
            "Ġmodel : 2746\n",
            "Ġfor : 329\n",
            "Ġcell : 2685\n",
            "Ġmigration : 13472\n",
            "Ġin : 287\n",
            "Ġpresence : 4931\n",
            "Ġof : 286\n",
            "Ċ : 198\n",
            "two : 11545\n",
            "Ġexternal : 7097\n",
            "Ġfactors : 5087\n",
            "Ġboth : 1111\n",
            "Ġinfluencing : 32596\n",
            "Ġcell : 2685\n",
            "Ġpolarization : 42704\n",
            ": : 25\n",
            "Ġcontact : 2800\n",
            "Ġguidance : 11154\n",
            "Ġand : 290\n",
            "Ġchem : 4607\n",
            "ot : 313\n",
            "axis : 22704\n",
            ". : 13\n",
            "ĠWe : 775\n",
            "Ċ : 198\n",
            "pro : 1676\n",
            "pose : 3455\n",
            "Ġtwo : 734\n",
            "Ġdifferent : 1180\n",
            "Ġsensing : 34244\n",
            "Ġstrategies : 10064\n",
            "Ġand : 290\n",
            "Ġwe : 356\n",
            "Ġanalyze : 16602\n",
            "Ġthe : 262\n",
            "Ġtwo : 734\n",
            "Ġresulting : 7186\n",
            "Ġmodels : 4981\n",
            "Ġby : 416\n",
            "Ġrecovering : 20222\n",
            "Ċ : 198\n",
            "the : 1169\n",
            "Ġappropriate : 5035\n",
            "Ġmacro : 15021\n",
            "sc : 1416\n",
            "opic : 16603\n",
            "Ġlimit : 4179\n",
            "Ġin : 287\n",
            "Ġdifferent : 1180\n",
            "Ġregimes : 25879\n",
            ", : 11\n",
            "Ġin : 287\n",
            "Ġorder : 1502\n",
            "Ġto : 284\n",
            "Ġsee : 766\n",
            "Ġhow : 703\n",
            "Ġthe : 262\n",
            "Ġsize : 2546\n",
            "Ġof : 286\n",
            "Ġthe : 262\n",
            "Ġcell : 2685\n",
            ", : 11\n",
            "Ċ : 198\n",
            "with : 4480\n",
            "Ġrespect : 2461\n",
            "Ġto : 284\n",
            "Ġthe : 262\n",
            "Ġvariation : 12291\n",
            "Ġof : 286\n",
            "Ġboth : 1111\n",
            "Ġexternal : 7097\n",
            "Ġfields : 7032\n",
            ", : 11\n",
            "Ġinfluences : 16717\n",
            "Ġthe : 262\n",
            "Ġoverall : 4045\n",
            "Ġbehavior : 4069\n",
            ". : 13\n",
            "ĠMoreover : 10968\n",
            ", : 11\n",
            "Ċ : 198\n",
            "we : 732\n",
            "Ġintegrate : 19386\n",
            "Ġnumer : 5470\n",
            "ically : 1146\n",
            "Ġthe : 262\n",
            "Ġkinetic : 37892\n",
            "Ġtransport : 4839\n",
            "Ġequation : 16022\n",
            "Ġin : 287\n",
            "Ġa : 257\n",
            "Ġtwo : 734\n",
            "- : 12\n",
            "dimensional : 19577\n",
            "Ġsetting : 4634\n",
            "Ġin : 287\n",
            "Ġorder : 1502\n",
            "Ċ : 198\n",
            "to : 1462\n",
            "Ġinvestigate : 9161\n",
            "Ġqual : 4140\n",
            "itatively : 48668\n",
            "Ġvarious : 2972\n",
            "Ġscenarios : 13858\n",
            ". : 13\n",
            "Ċ : 198\n",
            "Key : 9218\n",
            "word : 4775\n",
            ". : 13\n",
            "ĠKin : 16645\n",
            "etic : 5139\n",
            "Ġequations : 27490\n",
            ", : 11\n",
            "Ġmult : 1963\n",
            "isc : 2304\n",
            "ale : 1000\n",
            "Ġmodeling : 21128\n",
            ", : 11\n",
            "Ġmulti : 5021\n",
            "- : 12\n",
            "cue : 15509\n",
            ", : 11\n",
            "Ġnon : 1729\n",
            "- : 12\n",
            "local : 12001\n",
            ", : 11\n",
            "Ġhyd : 7409\n",
            "rod : 14892\n",
            "ynamic : 28995\n",
            "Ġlimit : 4179\n",
            ", : 11\n",
            "Ċ : 198\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}